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Letter from Director Moss and 
Superintendent Reykdal 


We share a passion for improving outcomes for youth in Washington state, and we know that, 
over time, we are serving the same children and families. Yet we work in systems that are often 
directed by artificial constraints such as ages, funding streams, and eligibility criteria. That 
means that we need to be vigilant about aligning our resources, finding efficiencies, and ensuring 
that Washington continues to build an equitable system for children, youth, and families that 
embraces the whole child. 


For example, 36 percent of children utilizing Working Connections Child Care Subsidy (more 
than 19,000 children) are school-aged. These children participate in many systems, including 
K-12, licensed before and after school child care, 21st Century Community Learning Centers 
and/or youth development programs. All of these programs play an important role in supporting 
Washington’s youth to become engaged citizens, active learners, and productive community 
members. 


Research clearly demonstrates that youth outcomes are tied to the quality of the programs in 
which they participate. Washington’s Legislature understands this—they’ve provided funding for 
K-12, Early Achievers, and expanded learning opportunities programs to support high-quality 
services for children and youth. As state agencies, we continue to learn a lot about the most 
effective ways to support quality improvement efforts. 


The Expanded Learning Opportunities Quality Initiative is a great example of state agencies, 
community partners, non-profits, funders, schools, researchers and direct service organizations 
working together to learn more about how to improve quality and support the youth in our state, 
and we are pleased to co-present this report on the results. ELO programs often serve as the 
“glue” for children and families, complementing school experiences, supporting working families, 
and engaging children and youth to discover their potential. We look forward to continuing to 
work together to ensure that these programs offer the highest quality to the youth in our state. 


Sincerely, 
Sx fe (boi P Ss Royle 
Heather Moss Chris Reykdal 
Director Superindendent of Public Instruction 
Department of Early Learning Office of Superintendent of Public Instruction 
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Executive Summary 


Expanded Learning Opportunities Quality Initiative 


Washington is building a comprehensive learning system 
that emphasizes quality from birth to 18 years. With $1 
million in funding from the Early Start Act, and matching 
funds from the Raikes Foundation, the Department of 
Early Learning (DEL) and the Office of Superintendent of 
Public Instruction (OSPI) have completed an Expanded 
Learning Opportunities Quality Initiative (“initiative”), 
piloting a high-quality, equitable system that supports 
providers of Expanded Learning Opportunities (ELO) 
with the training and resources needed to provide 
high-quality programs for youth after school and in the 
summer. 


Fifty programs from four counties (King, Pierce, Spokane, 
and Walla Walla) participated in a quality assessment 
and improvement process, including licensed family 
homes and child care centers (who serve school-age 
children and families) already rated in Early Achievers; 
federally funded 21st Century Community Learning 
Centers; and both licensed and unlicensed ELO 
programs serving youth between the ages of 5-18 years. 


Summary of Evaluation Findings 


Conducted by the University of Washington’s Cultivate 
Learning, the evaluation examined ELO program quality 
and the impact of improvement efforts. It also examined 
the experiences of participating coaches and providers 
to hear their perspectives on improvement interventions. 
Key findings include: 

¢ Programs initially assessed as lower than average 

improved after training and coaching. 


“I’ve really learned work ethic. I’ve learned how to 
* Coaching with fidelity to the model was linked with manage my time, how to prioritize. These were all 


sai setae pumas — skills that | was very bad at before | started that 

. igher levels of student engagement and fewer P . . 

ehallendine behaviors were «cen 2S program internship. And | feel very confident that | can go 
quality improved. into any job now and apply the same methods, same 

* Program staff and directors found coaching strategies that | did here and I’ll have no problem 


and data helpful and desired more frequent 
observations and feedback. 

¢ Programs experienced several systemic barriers - Youth ELO Participant 
to improving quality including unstable staffing, 
limited time for training and limited funding to 
support staff. 


succeeding in the work place.” 
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Initiative Goals 


Building upon a decade of quality improvement work in 
Washington, the initiative launched during the 2016-17 
school year to show that, with the proper support, 

ELO programs can deliver the type of high-quality 
engagement proven to result in meaningful youth- 

level outcomes. The initiative also aimed to create a 
streamlined system of supports including coaching and 
training for program staff to engage children and youth 
with developmentally appropriate strategies across 

the age span. Lastly, the initiative aimed to connect 
“systems,” specifically the early learning and school-age 
systems. 


Recommendations and Next Steps 


The Washington State Legislature appropriated 

$750,000 in the 2018 budget to continue the initiative 

an additional year. This state funding will continue 

to be matched with private funding from the Raikes 

Foundation. Priorities include: 

¢ Continue to support original participants: current 

participants will be given top priority for continuing 
their quality improvement work, and it’s expected 
that some new program participants in King, 


Pierce, Spokane and/or Walla Walla counties will Initiative Partners 
be added. 

¢ Create a pipeline for quality improvement: with Department of Early Learning 
various sizes, types, and existing levels of capacity Office of Superintendent of Public Instruction 
and quality, supports need to be created that foster School’s Out Washington 
quality improvements for all types of programs. University of Washington 

¢ Focus on coaching: coaching is critical to improving Child Care Aware of Washington (including 
program quality and it’s important coaches are well Child Care Resources and Community Minded 
trained, have enough time with programs, and can Enterprises) 
coach to fidelity standards. Raikes Foundation 


iad ? Additionally, community coalitions met regularly 
Youth Participant Demographics throughout the pilot year to discuss how to best meet 
the needs of youth and families in their region. 


0.1% 


BB White 
Mi Asian Program Site Locations 
© Hispanic 

HB American Indian 
@ Black 

I Pacific Islander 
© Other 


e 55.5 percent of youth participate in the Free/ 
Reduced Price School Lunch Program 

¢ 10 percent of youth are English Language learners 

e¢ 12.7 percent are youth with special needs 
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Overview of the Initiative 


Initiative Goals 


Washington is building a comprehensive learning 
system that emphasizes quality from birth to 18 years. A 
statewide quality early learning system has been scaled 
with support of the historic 2015 Early Start Act. The 
Early Start Act expands upon early learning to include 
piloting a high-quality, equitable system for school- 

age children to explore how best the state can support 
providers of Expanded Learning Opportunities (ELO) with 
the training and resources needed to encourage high- 
quality programs. Including afterschool and summer 
programs, high-quality ELO programs have been 

shown to improve youth outcomes, including academic 


improvements, social/emotional gains, and skill building. 


Launched as the Expanded Learning Opportunities 
Quality Initiative (“initiative”), the Department of Early 
Learning and the Office of Superintendent of Public 
Instruction convened a Steering Committee made up 
of providers, funders, and experts to provide direction 
on the design, implementation, and evaluation of the 
initiative. Building upon a decade of quality improvement 
work in Washington, the Expanded Learning 
Opportunities Quality Initiative launched during the 
2016-17 school year in youth development programs 
with a commitment to equity and reaching underserved 
populations. 


The goal of this initiative was to show that, with the 
proper support, ELO programs can deliver the type of 
high-quality engagement proven to result in meaningful 
youth-level outcomes. The initiative also aimed to create 
a streamlined system of supports including coaching 
and training for program staff to engage children and 
youth with developmentally appropriate strategies 
across the age span. Lastly, the initiative aimed to 
connect “systems,” specifically the early learning and 
school-age systems. By leveraging resources across 
these two systems (for example, coaching and training), 
Washington can improve the quality of both systems and 
provide more seamless services to children, youth, and 
families. With the 2017 creation of the new Department 
of Children, Youth, and Families (DCYF), creating efficient 
systems for supporting children and youth across the 
age spectrum is increasingly vital. Lessons learned 
throughout the initiative are providing information about 
ways to connect the systems and supports that promote 
quality from birth through age 18. 


Initiative Overview 


The initiative was designed as a learning pilot, building 
on current quality improvement efforts, and informing 
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how the state will support ELO in the future. In the 
spring of 2016, the Department of Early Learning (DEL) 
and the Office of Superintendent of Public Instruction 
(OSPI) worked with School’s Out Washington, a non- 
profit statewide intermediary, to recruit qualified 
programs to volunteer for the initiative. Once selected, 
programs received training, coaching and assessment 
using the best national research on promoting positive 
youth outcomes. During this period, the University of 
Washington conducted a rigorous evaluation to assess 
the progress toward meeting the initiative goals. In 
addition, staff, families and youth were interviewed 

for perspective on their respective ELO programs. The 
results of this initiative are captured in this report. 

Fifty programs from four counties (King, Pierce, Spokane, 
and Walla Walla) were chosen to participate in a quality 
assessment and improvement process. Participating 
programs included licensed family homes and child care 
centers (who serve school-age children and families) 
already rated in Early Achievers; federally funded 21st 
Century Community Learning Centers and both licensed 
and unlicensed ELO programs serving youth between 
the ages of 5-18 years. The initiative aimed to reach 
children and youth where they were already being served 
in a mix of program types in order to learn as much as 
possible about supporting quality improvement efforts in 
ELO programs. 


A Learning Initiative 


The ELO initiative was designed to include a rigorous 
evaluation component to serve two overarching 
purposes: 
1) To understand quality in a sample of ELO sites and 
to explore a quality improvement intervention; and 
2) To understand the experience of coaches and 
providers in the pilot and hear their perspectives on 
improvement interventions. 


Steering Committee 


Department of Early Learning (DEL): oversight of the 
state’s Quality Rating & Improvement System (QRIS), 
Early Achievers, and co-leader of the Expanded 
Learning Opportunities Quality Initiative. 


Office of Superintendent of Public Instruction 
(OSPI): agency that has oversight of 21st Century 
Community Learning Centers and co-leader of the 
Expanded Learning Opportunities Quality Initiative. 


School’s Out Washington: statewide intermediary 
leading and coordinating the quality initiative 
supports and services for ELO programs across the 
state. 


University of Washington (UW): evaluators of the 
initiative and developers of the statewide coaching 
framework, professional development and online 
coaching platform. 


Child Care Aware: coaching partner for initiative 
sites that are also participating in Early Achievers. 


Raikes Foundation: philanthropic partner that has 
funded quality improvement in ELO since 2009. 


Each purpose was comprised of multiple research 
questions and required a unique study design and 
methodology. In the following section, there is an 
overview of the evaluation including the timeline, 
participant recruitment and selection, measures, and 
a description of the evidence-based coaching model as 
well as a description of each of the studies. 


Funding 


The Washington Legislature appropriated $1M to 
support the Expanded Learning Opportunities Quality 
Initiative for planning and one year of implementation. 

In order to Support a more robust evaluation, deeper 
community outreach, and the development of systems 
supports such as data, the Raikes Foundation provided a 
private match to the state investment for the initiative. 


Why Expanded Learning Opportunities 


Access to quality ELO programs complements 
classroom instruction. Research shows that quality 
ELO programs improve attendance and grades while 
building the social-emotional skills needed to succeed 
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in school and in life. Reflecting the needs of their local 
communities, these youth-driven learning opportunities 
are particularly powerful for historically underserved 
populations. They engage and inspire young people 

to explore their passions, develop their talents and 

find their unique voice as future leaders. Based on 

a national body of research, the framework of this 
initiative assumes that high-quality programs lead to the 
following youth outcomes: 


Academic Benefits: 
¢ Decreases in absences during the school day 
¢ Decreases in challenging behavior (misconduct, 
aggression, etc.) 
¢ For academically based programs, increases in 
math and literacy scores 
e Increases in grade point average 


Social Emotional Learning Benefits: 

e Increases student engagement, intrinsic 
motivation, concentrated effort and growth 
mindset 

e Increases emotional regulation and cooperative 
skills 


Career Pathway Skills/Skill Development 
¢ Specific skill development tied to program content 
(e.g., Arts, Media, STEM related skill building) 
¢ Increases in initiation and leadership abilities 


In addition to these outcomes, other associated benefits 
of ELO programs include providing a safe space for 
adolescents (and its associated decreases in juvenile 
delinquency) as well as a support for working families. 


Background on Expanded Learning 
Opportunities in Washington 


Washington is working to support youth across the state 
access to high-quality learning opportunities beyond 
the classroom, and is leading the nation in supporting 
high-quality early learning and sustaining those gains 
into elementary and beyond. The ELO Quality Initiative 
builds on Washington’s work to promote quality in youth 
development programs using the David P. Weikart’s 
Youth Program Quality Intervention (YPQI) to support 
and assess quality in partnership with School’s Out 
Washington. This has led to large scale adoption of a 
continuous quality improvement intervention in the 
state with over 500 programs participating in the quality 
intervention statewide. The initiative built upon this prior 
work, incorporating valuable lessons learned about 
scaling high-quality practices to ELO programs across 
the state. 


Early Start Act Enabling Language 


“By July 1, 2017, the department [of early learning] 
and the Office of Superintendent of Public Instruction 
shall jointly design a plan to incorporate school-age 
childcare providers into the early achievers program 


or other appropriate quality improvement system. To 
test implementation of the early achievers system 
for school-age childcare providers the department 
and the Office of Superintendent of Public Instruction 
shall implement a pilot program.” 


Recently, the Sparkwind 
Movement launched in 
Washington, a campaign 
aimed at ensuring all 
young people have 
equitable access to high- 
quality ELO programs. 
The movement is intended to build on current efforts 
and garner collective impact by building a high-quality 
system of ELO programs for youth. The Sparkwind 
Movement is intended to connect all of the existing work 
underway in Washington to promote high-quality ELO 
programs for youth under one umbrella. 


seg MOVEMENT 


The Sparkwind Movement is currently led by School’s Out 
Washington and has been informed by a broad sector of 
stakeholders in the afterschool and youth development 
field, including: youth serving organizations; public sector 
partners; and youth who participated in focus groups to 
help develop the core values and language to define the 
movement. The Sparkwind Movement name was inspired 
by the stories and voices of youth in Washington. 
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Results and Lessons Learned: 
Evaluation 


Overview of the Intervention 


Fifty programs from four counties (King, Pierce, Spokane 
and Walla Walla) participated in the initiative including 
licensed family homes and child care centers (who serve 
school-age children and families) already rated in Early 
Achievers; federally funded 21st Century Community 
Learning Centers and both licensed and unlicensed 

ELO programs serving youth between the ages of 5-18 
years. These programs received training, professional 
development opportunities and weekly on-site practice- 
based coaching. The evaluation conducted ongoing data 
collection throughout the intervention to assess the 
impact of the intervention on program quality. 


Program Selection and Recruitment 


The Expanded Learning Opportunities Quality Initiative 
Steering Committee established selection criteria for 
types of programs to participate in the study including 
the length of time in operation, licensing standing, prior 
engagement in quality improvement efforts, leadership 


Overview of Participants 


Location 

King County: 16 
Walla Walla County: 7 
Pierce County: 13 
Spokane County: 14 


Program Type 

21st Century CLCs: 12 

Licensed Family Homes (FCC): 5 
Licensed Centers: 7 

Youth Development Programs: 11 
Licensed School-Age Child Care: 15 


support and number of youth participating in the 
program. All program types (licensed family homes and 
childcare centers, 21st Century Community Learning 
Centers, and school-age and youth development 
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programs) that fit the established criteria and were 
within the specified regions were recruited to participate 
in the evaluation. Organizations that met the criteria 
were proactively contacted to assess their interest in 
participating. The final selection of sites included a 
review by the evaluation team to ensure that the sites 
represented enough regional and programmatic diversity 
to support the study design. 


Programs were offered incentives to participate including 
20 to 30 hours of training and professional development 
tied to the School-Age and/or Youth Program Quality 
Assessment (SAPQA or YPQA, referred to collectively as 
the PQA) valued at $7,250, and stipends that ranged 
from $1,500 to $3,000 depending on the size of the 
program. Once applications were submitted, programs 
were also awarded a signing bonus of $300. 


Youth Participants 


Approximately 1,100 youth participated in the initiative, 
and through the evaluation process, many youth program 
participants were able to be matched in the Educational 
Research and Data Center (ERDC) data system. With 
parent consent, and using first and last name, grade, 
and name of school approximately 70 percent of youth 
participating in initiative sites were able to be matched 
in the ERDC data system. This has provided information 
on the youth currently engaged in pilot programs. 

e 55.5 percent of youth participate in the Free/ 

Reduced Price School Lunch Program 
¢ 10 percent of youth are English Language learners 
e 12.7 percent are youth with special needs 


Coaching 


Youth Participant Demographics 


BE White 

BB Asian 

{)) Hispanic 

HB American Indian 
@ Black 

BB Pacific Islander 
0) Other 


School’s Out Washington (SOWA) and Child Care 
Aware of Washington (CCA) were responsible for the 
recruitment and supervision of coaches. The majority 
of the fourteen coaches were selected from an existing 


“Through action- planning and observations, | 
watched these teachers grow more confident 
and independent as their skills and strategies 
developed with a new sense of accountability for 
quality-improvement.” 

- Monika Egland, Early Achievers/ELO Coach 


pool of coaches with 5-25 years of experience in 
coaching, education, and/or youth development. The 
licensed childcare centers and family child care homes 
selected to participate in the initiative were involved with 
Early Achievers, and were assigned the same coach for 
consistency. 


Coaches were expected to meet with their programs 
weekly for the duration of the study, conducting in- 


Supporting 


High-Quality 
Programs 


person coaching sessions every other week, and online 
coaching sessions via Coaching Companion (an online 
coaching platform) on alternating weeks. All in-person 
and online coaching was modeled after the evidenced 
based Practice-based Coaching framework, in which 
coaches worked with programs to create an action plan 
with set goals, conducted focused observations aligned 
to those goals, and provided feedback and reflection 
opportunities based on the observed practice. An 
overview of Practice-based Coaching can be found in the 


appendix. 


Overview of the Evaluation Methodology 


Conducted by the University of Washington’s Cultivate 
Learning, the overarching goal of the initiative evaluation 
was to examine ELO site quality and impact of 
improvement efforts, such as practice-based coaching, 
in ELO programs. The evaluation was approved by the 
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University of Washington Human Subjects Institutional 
Review Board in July 2016. 


The evaluation spanned 34 weeks and included a pre- 
assessment, an ongoing improvement intervention led 
by SOWA and CCA of WA coaches and trainers, anda 
post-assessment. The improvement intervention was 
started in a staggered fashion across five cohorts. Each 
program received five weeks of baseline quality data 
collection, followed by the intervention for the remaining 
29 weeks, until the post assessment. Each week all sites 
videotaped one hour of programming, and submitted 
their videos to Cultivate Learning for coding and analysis. 
When trends were identified they were shared with 
SOWA, along with suggested improvements, for coaches 
to use in the intervention. 


Focus groups were also conducted with participating 
programs and with coaches. The focus groups were 
facilitated to learn from coaches, program staff and 
leadership about their experiences in the initiative. 
Individual interviews were conducted with youth 
participants to capture their feedback as well. 


Purpose of the Evaluation 


The evaluation was designed to serve two overarching 
purposes: 
1) To understand quality in a sample of ELO sites and 
to explore a quality improvement intervention; 
2) To understand the experience of coaches and 
providers in the pilot and hear their perspectives on 
improvement interventions. 


Each purpose was comprised of multiple research 
questions and required a unique study design and 
methodology. In the following section, there is an 
overview of the two studies conducted by the University 
of Washington. The full evaluation report can be found in 
the appendix. 


Multiple Baseline Approach 


This evaluation required a study design that would 
allow for continuous quality improvement throughout 
the initiative. A multiple baseline approach was 
chosen to allow for data to be collected and analyzed 
frequently, identify program elements that support 
or undermine improvement efforts, and allow 

for responsive changes to be made that improve 
effectiveness. In other words, this method made 

it possible to adjust and revise the intervention in 
response to what is and is not working. This design 
attempts to control for the effect of extraneous events 
by showing that specific changes are associated with 
the onset of the intervention. 
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Study #1: Program Quality and Practice- 
based Coaching 


Purpose: to understand quality in a sample of ELO sites 
and to explore a quality improvement intervention. 


Research Questions 


e What is the baseline quality of programs 
included in the sample? 

e How does Practice-based Coaching 
work across multiple settings, and what 
refinements are necessary to improve 
practice? 

¢ How do programs change over the course of 
the initiative? 

e How is the intervention effect mediated by 
coaching hours and fidelity? 


Summary of Findings 


Programs that were assessed as having lower 
than average quality scores before coaching 
improved in some domains of quality after 
training and Practice-based Coaching. 
Coaches need ongoing support and 
incentives to implement practice-based 
coaching in person and online. 

Fidelity to the coaching model and the 
number of coaching hours are related to 
program improvement. 


¢ Coaching with fidelity to the Practice-based 
Coaching framework showed improvement on 
program quality. 

e Higher levels of student engagement and 
fewer challenging behaviors were seen as 
program quality improved. 


In the chart on the right, quality 
levels are tracked during 

three phases 1) baseline data 
collection, 2) during staff training, 
and 3) during the period of the 
coaching intervention. The level 
of quality may vary each week, 
but the data show that the trend 
is for quality to improve after 
training occurs and it continues 
to increase throughout the 
intervention with practice-based 
coaching. 


Example of Coaching Intervention to Improve Quality 


Measure of quality 


123456789 0 1 12 128 14 14 14 17 #18 19 20 21 


Weeks of intervention 
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Increases in Quality Scores throughout the 
Intervention 


Using the School-Age and/or Youth Program Quality 
Assessment (PQA), participating programs showed 
improvement in program quality over the course of 

the intervention. The intervention was focused on 
improvements in Interaction and Engagement, the 
domains most connected to youth outcomes. These two 
domains demonstrated the strongest growth through the 
intervention. Most programs entered the intervention 
already strong on safe and supportive environments. 


PQA Pre and Post Assessment for 50 Programs by Domain 


PQA Measure 


Safe Supportive Interaction Engagement 
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Study #2: Stakeholder Experiences 


Purpose: to understand the experience of coaches and 
providers in the pilot and hear their perspectives on 
improvement interventions. 


Research Questions 


e What were the experiences of site staff, 
leadership and coaches that participated in 
the initiative? 

e What are the perspectives of site staff, 
leadership, and coaches regarding coaching, 
professional development, and overall 
improvement efforts that could contribute to 
a more effective intervention design? 


Additional Research and Evaluation 


Conducted by American Institute for Research (AIR), 

an additional evaluation is underway to assess the 
impacts of Washington’s 21st Century Community 
Learning Centers (21st CCLC). There are 141 21st 

CCLC in Washington, and 11 of these programs also 
participated in the initiative. In order to assess the 
manner in which youth may have changed during their 
enrollment in 21st CCLC-funded programs during the 
span of the 2016-17 school year, the Youth Motivation, 
Engagement, and Beliefs Survey was used on a pre-post 
basis to collect data on a variety of youth development- 
related outcomes, including academic identity, positive 
mindsets, self-regulation, and interpersonal skills. A 
total of 346 youth in grades 4 through 9 from the 11 
21st CCLC-funded sites took the pre-survey between 
early October and mid December 2016, while post- 
survey data was collected from 282 of these same youth 
between late April and early June 2017. 


Early results show that youth entering programs with 
lower baseline scores showed significant growth 
across each of the four survey constructs - academic 
identity, positive mindsets, self-management, and 
interpersonal skills. In particular, gains were especially 
pronounced among youth with higher levels of 21st 


Summary of Findings 


e PQA training helped program staff to define 
and “see” quality. 

¢ Program staff and directors found coaching 
and data helpful, and desired more frequent 
observations and feedback. 

e When coaches were able to include program 
leadership, sites were able to collaborate 
more and resolve issues as they arose. 

¢ Online coaching was regarded by program 
staff and coaches as having potential 
but more support is required for coaches 
and program staff to use the Coaching 
Companion. 

e¢ Programs experienced several systemic 
barriers to improving quality including 
unstable staffing, limited time for training and 
limited funding to support staff. 


What is a Quality Seal 


Programs in Washington have been using the PQA 
to assess ELO program quality levels for youth 
services that are safe, supportive, interactive, and 
engaging. The Quality Seal is an indication that 
programs have attained a threshold of quality in 
similar areas and are offering a level of quality 
that is commensurate with youth outcomes. 
Developing and validating a Quality Seal was a 
goal of the initiative. When finalized, Washington 
will be the first state in the country to have valid 
tools for certifying quality in ELO programs across 
all settings. For more information on the Quality 
Seal, see the appendix. 


CCLC participation (60-day or more attendance) for 
the academic identity, positive mindsets, and self- 
management constructs. Centers with higher scores 
on the Youth Program Quality Assessment have higher 
levels of youth-reported engagement on average. 


A full evaluation report on the 21st CCLC is expected in 
October 2017. 
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WAAL 


Results and Lessons Learned: 


Implementation 


In addition to the evaluation results on program quality 
improvement, much was learned during the ELO Quality 
Initiative pilot about implementing future efforts to 
improve the quality and access of ELO programs. As 

a pilot, the implementation was built to garner as 

much learning as possible to allow for streamlined 
implementation in the future. Highlights of these 
implementation lessons learned include: 


High Program Demand: there was strong interest from a 
diverse group of ELO providers to participate in the ELO 
pilot. Recruitment was successful across all program types 
and there was very low turn-over during the pilot year. 


Importance of Strong Partnerships: the Steering 
Committee demonstrated flexibility and a willingness 

to work together in support of the pilot. Representing 
21st Century programs, School Age Centers, Youth 
Development Programs, and Licensed Child Care Centers 
and Family Homes, the functionality of the Steering 
Committee allowed for quick and collaborative decision 
making. 


Benefit of Private, Complementary Funding: private 
funding was available to support several additional 
elements of the pilot that proved essential including: 

e Expanded evaluation allowed for a process and 
impact study, enabling a deeper evaluation that 
allowed for “real time” quality improvements. 

e Additional training for pilot sites. 

¢ Development of the Youth Program Registry (see 
side bar on page 18). 


Balance Between Evaluation Requirements and 

Program Needs: developing and maintaining fidelity to 

an evaluation can be both supportive and challenging to 
implementation. The evaluation data collection allowed for 
quick identification of problems and enabled immediate 
troubleshooting. Information could be shared quickly with 
coaches to mitigate potential issues. However, programs 
often struggled to ramp up new program requirements 
while simultaneously meeting the evaluation requirements, 
such as weekly videotaping. The staggered timeline was 
challenging for participants that started later in the project 
and did not receive support services until late in the year. 


Expanded Learning Opportunities Quality Initiative | 17 


Start-Up Challenges: like any new initiative, the ELO pilot 
struggled with several start up issues, stemming in large 
part from a truncated timeline. While these problems 
could be frustrating for partners and participants alike, 
they were neither unexpected nor unique to this pilot. 
More upfront planning timing would have been welcome, 
particularly in developing clear communications strategies 
between partners and participants. While these issues 
often were at the forefront of implementing the pilot, most 
of these issues were solved through implementation. 


Important Connection Between ELO and Early Learning: 
the pilot was designed using lessons learned from the 
implementation of Early Achievers, and provided a unique 
opportunity to create an intentional bridge between early 
learning programs and ELO programs. In fact, many 
programs already serve youth of both age groups. The 
licensed child care centers and family homes in the pilot 
were Early Achievers sites and participation gave them 
the opportunity to integrate quality improvements across 
all ages. In addition, several coaches worked across 
Early Achievers and the ELO pilot. The experiences using 
Practice-based Coaching in Early Achievers provided 
valuable insight to inform this initiative; and lessons 
learned from this initiative are currently informing Early 
Achievers. The initiative highlighted the benefit to both 
systems of working in alignment. 


Engagement Needed at Site Level and Administrative 
Level: larger organizations with multiple sites required 
unique approaches to engage both the organizational 
leaders and the site level leaders. For example: 
¢ Organizational level leadership needs to be 
supportive and to demonstrate buy-in through 
“sponsorship from the top,” and Director support 
was important. 
¢ Engagement at the administrative/supervisor level 
was Critical for site level success However, follow- 
through and commitment was strongest when 
recruitment was done at the site level, rather than 
through the administrative level. 
e Site level staff turnover was a challenge throughout. 


Different Participant Types have Unique Needs: program 
participants in the pilot were very diverse and included 
large and small youth-serving organizations, school 
based programs, licensed child care centers and family 
homes. Each of these program types had different 
support needs and required tailored approaches to 
service delivery. For example: 
¢ Family Homes need alternate scheduling options 
for training (e.g. evening and weekends). 
¢ Organizations with multiple sites need different 
communications channels to reach both site level 
leaders and organizational level leaders. 


Washington Youth Program Registry 


Included in the pilot was the development of 

a new Youth Program Registry, a free tool and 
comprehensive database to support ELO programs, 
youth development professionals, and quality 
information for Washington state. Built with private 
funding, this is the first registry of its kind and the 
data collected will provide valuable information to 
programs, state agencies, and eventually parents 
about how to best serve Washington’s youth. Lesson 
learned in developing the registry include: 

¢ Having the field help design the registry was 
critical. It provided quality assurance to the 
design, recognized the importance of the 
end user, and encouraged participation after 
launch. Creating a unified database has had 
universal support. 

¢ Reports help programs track organization, 
site, and/or program level data, enabling 
better decision making. It is particularly helpful 
for smaller programs by providing essential 
infrastructure Support. 

e The Registry provides state level aggregate 
data that advances the field and prepares for 
the future quality expansion efforts. 

¢ Building new tech platforms is challenging and 
a lot of planning is needed before moving to 
build out. 

e User testing is critical and planning for multiple 
rounds of user testing before launch is essential. 


More information on the Youth Program Registry can 
be found in the appendix. 


Value of Coaching Model: coaching is essential to 

quality improvements and there are ways to ensure 

that coaching is deployed effectively. For example: 

¢ Full-time coaches are able to best support programs 
as they need to learn and master the coaching model 
and have time for consistent work with sites. 

¢ Aminimum amount of coaching per site needs to 
be dedicated and planned for at the program level. 

¢ Cross training with Early Achiever coaches and ELO 
coaches was effective. Early Achievers coaches 
have expertise in practice-based coaching and 
ELO coaches have expertise in the Youth Program 
Quality intervention and serving ELO programs. 

¢ Coaching should include working with site and 
organization leaders as well as deep work with 
observation, reflection and modeling with adults 
working with youth directly. 
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Community Feedback on System 
Needs 


Engaging participating communities has been an 
essential part of the ELO pilot. In addition to having a 
“Community Hub” to help support local implementation 
of the pilot, participating communities received funding 
from the Raikes Foundation to create “Community 
Coalitions,” locally facilitated groups of ELO programs, 
participating youth, and community leaders. These 
groups met regularly throughout the pilot year to discuss 
how to best meet the needs of youth and families in their 
region. 


Community Coalitions facilitated community 

focus groups to learn more about the needs of 

youth and families to inform this report and make 
recommendations about how to build an ELO system 
that supports strong outcomes for youth and aligns 
with local priorities. Focus groups were conducted in 
King, Pierce, and Spokane counties and included a 
series of quantitative and qualitative questions that 
are summarized below. Additional interviews were 
conducted with programs, staff, coaches, and youth in 
all four counties. Full focus group reports for the three 
communities are provided in the appendix. 


“I’m better at listening to people. | think that 
before when people talked to me, even a family 
member, | just kind of listened to the parts | 
wanted to hear and besides that | just didn’t care 
for the rest. But now I’m listening to everything and 
| understand better.” 

- ELO Youth Participant 


Demand for ELO Services 


Focus groups all agreed that there is a high demand 

for ELO services in their community and not enough 
programs to meet the demand. In addition, focus group 
members highlighted that youth-serving programs want 
access to quality improvement supports like training and 
technical assistance. 
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Current Barriers for Youth and Families 


¢ Transportation: many children and youth lack 
access to reliable transportation, creating barriers 
to regular attendance in ELO programs. 

¢ Cost: fee-based programs are cost prohibitive for 
families and there are not enough tuition-free 
programs to meet the demand. 

¢ Language and cultural responsiveness: programs 
need to be accessible to families, target outreach 
appropriately, and have resources available in 
families’ home languages. 


Challenges for Youth-Serving Programs 


¢ Staffing: attracting and retaining highly qualified 
staff remains a challenge for ELO programs. Low 
wages coupled with a lack of full-time employment 
(most ELO positions are part-time, serving 
children and families before and after-school or 
in the summer) pose challenges to creating and 
maintaining a diverse and qualified workforce. 

¢ Need for capacity building: smaller organizations 
are important to serving diverse communities 
and often need additional organizational and/or 
financial support. 

¢ Supports for quality improvements: programs want 
access to quality improvement supports that are 
flexible, affordable, and meet the specific needs of 
their program. 

¢ Sustainability: ongoing funding to support high- 
quality ELO programs is a struggle for many 
programs. 

¢ Connection with schools: partnerships between 
schools and ELO programs leverage additional 
resources (e.g. space) and allows for aligned goals 
and supports to improve student outcomes and 
data sharing to track those outcomes. 


Importance of Youth-Serving Programs in 
Communities 


Focus group participants highlighted the need for year- 
round, financially accessible programs for youth in their 
communities. These programs need to: 
¢ Focus on diversity and equity. 
¢ Support social-emotional growth. 
¢ Support a well-qualified, diverse and appropriately 
compensated staff. 
e Utilize best practices through training, technical 
assistance and use of data. 


Focus groups spoke to the important role that high- 
quality ELO programs have in their communities to 
support youth and their families. 


94% of Coalition members agree or 
strongly agree that there is family demand for 
youth programs in their community. 


90% of Coalition members agree or 
strongly agree that with additional funding, 
youth-serving programs in their community 
would be interested in participating in quality 
improvement efforts like training and technical 
assistance. 


15% of focus group participants disagree or 
strongly disagree that there are enough youth 
serving programs in their community to meet 
the demand. 


“This program makes me feel welcome and safe 
because no matter how I’m feeling, what’s going 
on at home, what trouble | might have got into, | 
can come here... | know when | come here no one’s 
going to attack me based on my past, what I’ve 
done wrong, what’s not gone so well in my life.” 

- Youth ELO Participant 
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Youth Feedback 


A part of the ELO Quality Initiative evaluation was hearing 
from youth participants to get their perspective on ELO 
programs and what makes them engaging, interesting, 
and worthwhile to attend. Nineteen youth from eight 
participating programs were interviewed as a part of 

the University of Washington’s evaluation. Below is a 
summary of their perspectives. 


Skill building: several youth participants shared that they 
felt their program improved their social skills and grew 
their self-confidence. For example, one youth said, “I’ve 
really learned work ethic. I’ve learned how to manage 

my time, how to prioritize. These were all skills that | was 
very bad at before | started....| feel very confident that | 
can go into any job now and apply the same methods, 
same strategies that | did here, and I'll have no problem 
succeeding in the work place.” 


Idle hands: youth described how the program provided 
a positive alternative activity for them. If they were 

not attending an ELO program they would be home 
alone, watching TV, doing nothing or doing nothing but 
homework alone. Youth regarded their programs as 
keeping them engaged in positive social activity. One 
youth stated, “This program has saved me from a life 
of crime and doing things that | would not be proud of. 
Doing things | would not want my family to see. And this 


“Our program facilitator is helping me to get into 
a technical college so | can learn a trade craft. 
I’m studying electrical construction and | chose 
that because it’s a good, practical application 
that everyone here can help teach me about...I’m 
getting all kinds of educational help, different 
opportunities that | wouldn’t have had otherwise.” 
- ELO Youth Participant 


program means so much to me because | would be a 
totally different person without it and | would not be as 
successful today as | am without it.” 


Welcome and safe: youth described how their programs 
made them feel welcomed and safe. They felt supported, 
heard and mentored. For example, one youth offered, 
“This program makes me feel welcome and safe because 
no matter how I’m feeling, what’s going on at home, 
what trouble | might have got into, | can come here. | can 
talk to my youth mentors and | can share what’s going on 
and get very valuable insight on how to handle certain 
things that | may not be good at handling on my own.” 
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Recommendations 


Based on the results of the evaluation and 


implementation lessons learned, the Steering Committee 


of the Expanded Learning Opportunities Quality 
Initiative submits the following recommendations for 
consideration with future ELO Quality initiatives to 


ensure the highest system efficiency and strongest youth 


program outcomes. 
Scalable Quality Systems 


¢ Create a pipeline for quality improvement: 
ELO programs operate in various sizes, types, 
and existing levels of capacity and quality. 
Supports need to be created that foster quality 
improvements for all types of programs, which 
meet programs where they are and offer 
meaningful quality improvement paths. 


¢ State level infrastructure: state level capacity is 


needed to support scalable, statewide quality 


improvement efforts. The ELO field has benefitted 
from the engagement of multiple state agencies 


and organizations and a clear governance 
structure would further promote the field. 


¢ Connect early learning, K-12 and ELO: children 


and youth are best served when there are 
seamless transitions at critical developmental 
milestones. Continuing to align the ELO 


professional development and quality improvement 
system with the K-12 and early learning systems 
best meets the needs of children and leverages 
state and local resources. 

Community level engagement: ELO programs 

are uniquely suited to meet the local priorities 

of communities, families, and youth. State level 
efforts need to continue to engage with and build 
resources that fit community needs and honor 
local efforts. 


Coaching and Intervention Model 


Coaching is essential: coaching is critical to 
improving program quality and it is important that 
coaches are well trained, have enough time with 
programs, and can coach to fidelity standards. In 
coordination with Early Achievers, a designated 
coach is critical to engage the program staff and 
leadership in regularly scheduled trainings and 
coaching on program assessment, evidence-based 
goal setting, focused observation, and feedback, 
as well as trainings on research-based best 
practices in the ELO field. Frequent monitoring and 
feedback of coaching fidelity should be put in place 
to increase the likelihood of coaching impact on 
program quality. 
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¢ Link training and coaching: creating synergy 
between training and coaching maximizes the 
benefits of both and programs that received more 
access to coaching and training improved their 
quality overall. Ongoing access to professional 
development and training opportunities while 
the program is operating reduces the amount of 
outside training needed. Staff participating in 
training and professional development outside of 
program hours should be compensated for their 
time. 

¢ Coaching and training should be offered program 
wide: coaching should include all of the staff within 
the program, as well as the director. Disconnection 
between staff, coaches and leadership can 
interfere with improvement efforts. Conversely, 
when leadership and staff collaborate in coaching 
sessions, program interferences in improvement 
efforts could be resolved by the leadership. 
Allowing coaching to be inclusive across the 
program can help to align goals, and mitigate 
programs in quality improvement efforts. 

¢ Adequate time and intensity: improvement based 
interventions should have an adequate amount of 
time to ensure trainings and coaching is ongoing 
and producing quality improvements. 


Resources for Coaches and Programs 


¢ Coaching Companion: coaches and program 
staff and leadership can better utilize Coaching 
Companion, an online coaching tool that 
allows deeper coaching support to programs to 
complement in-person coaching. Accommodating 
for difficult scheduling times, Coaching Companion 
enables online reviews of performance by allowing 
program staff to capture and share practice for 
feedback. However, adequate training, technical 
improvements, and support need to be provided 
to coaches and staff for Coaching Companion to 
be successful. The costs and resources associated 
with supporting the use of Coaching Companion 
are far outweighed by the benefits of using the tool 


“Participating in this initiative allowed me to guide 
my Early Achievers Family Homes to evaluate 
and improve their care of school aged children. 
It validated that these sites were already offering 
high quality programming for their before and after 
school programs (in large part because of the work 
they have been doing with EA Quality Standards) 
and the goals they set for the pilot wound up also 
improving their Instructional Support practices.” 

- Andee, Early Achievers/ELO Coach 


as it uses fewer resources than in person coach 
visits and allows for specific program practices to 
be reviewed with intentional focus. 

¢ Resource Library: ELO programs benefit from 
having access to a resource library of videos, 
articles, and other resources that support the 
improvement effort. This allows program staff to 
learn and develop outside of program hours, and 
enables multiple staff and leadership to access 
the same materials and build consensus around 
aspects of quality and goals for improvement. 


Future Research and Evaluation 


¢ More research: more research is necessary to 
determine an adequate amount of evidence-based 
coaching and research-based trainings that leads 
to program improvement. In addition, continue to 
explore the relationship between quality, youth 
engagement, and youth outcomes. 

¢ Development of assessment tools: continued 
refinement and development of the evaluation and 
assessment tools will be necessary prior to broad 
statewide adoption. This includes the tools for 
measuring multiple aspects of ELO programs such 
as program quality, youth engagement, etc. 
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Next Steps 


The Washington State Legislature appropriated 
$750,000 in the 2018 budget to continue the Expanded 
Learning Opportunities Quality Initiative. This state 
funding will continue to be matched with private funding 
from the Raikes Foundation. While there are fewer 
resources available than in the initial pilot, there are 
several key priorities for continuing to support quality 
improvements in Washington ELO programs including: 


¢ Continue to support original participants: as 
quality improvement work typically takes more 
time than was possible in the original pilot, current 
participants will be given top priority for continuing 
their quality improvement work with targeted 
resources based on the pilot information. Due 
to some attrition, it’s expected that some new 
program participants in King, Pierce, Sookane and/ 
or Walla Walla counties will be added. 

¢ Coaching is key: coaching has been shown to be 
an essential element of quality improvement. In the 
upcoming year, there will be a continued effort to 
strengthen and expand practice-based coaching. 

¢ Refined evaluation: in the prior initiative, learning 
from the evaluation about the efficacy of the 
program model was essential. Now, the evaluation 
will shift to learning about the varied needs of 
programs to reach high thresholds of quality as 
well as the level and modality of coaching that is 
most effective. This means that there will be less 
need for participating programs to collect data, 
specifically videotaping, on site, making it easier 
for program participants. 

¢ Continued learning: there is a lot to learn about 
how to support quality improvement efforts in a 
variety of program types. ELO programs are diverse 
and there is not one single way to engage in quality 
improvement efforts. Rather, Supports will need 
to be tailored to meet the needs of youth serving 
organizations, particularly programs with limited 


Appendix 


“Because |’m a kind of person that, if | don’t do 
something right, then it’s just failure altogether. 
The project showed me progress that I’ve made and 
things are sometimes like, you start somewhere 
and you finish somewhere, it’s continuous, and you 
just have to keep trying at it.” 


- ELO Youth Participant 


capacity. Working in alignment with other local 
initiatives will help to better understand how to 
provide comprehensive supports for programs with 
operational needs in addition to quality supports. 

¢ Youth Program Registry: having access to 
program and youth level data is critical to making 
informed decisions at the state, community and 
program levels. The Youth Program Registry is 
already starting to have an impact, and additional 
build-out of the registry will allow for greater 
understanding of ELO strengths and needs. 

¢ The Sparkwind Movement: ELO programs across 
the state continue to engage in efforts to improve 
the quality of services available to youth under the 
Sparkwind Umbrella. The Sparkwind Movement 
builds on current efforts and garners collective 
impact by building a high-quality system of ELO 
programs for youth. The continuation of the pilot is 
aligned with this work to promote equitable access 
to high-quality ELO programs for youth. 

¢ Continue to engage youth: getting feedback 
directly from youth about ELO programs is 
essential. The Steering Committee will continue to 
explore ways to engage youth directly in program 
development. 


You can find the appendix referenced throughout this report at httops://www.sparkwindmovement. 
org/wp-content/uploads/2017/10/2017-ELO-Quality-Initiative-Report-Appendix.pdf. 
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